1. Field of the Invention
The present invention relates to a system and method for enhancing facial images by superimposing virtual objects on top of the continuous 2D human face image in real-time automatically and dynamically, using the facial feature information. The technique gives the user a personalized virtual reality experience overall.
2. Background of the Invention
There is a significant need in the art for a system and method for enhancing human facial images in the virtual world. Human face has been one of the most fascinating objects in computer vision. Capturing and modeling of the human face image is frequently attempted in virtual reality. For example, avatar creation is an interesting technique, which is also commercially used by many companies. The avatar creation tools usually superimpose the human face image onto the static template or pre-handled video image background. One good example of using the pre-handled video image sequences is suggested by S. Lee, K. Wohn, Y. Ahn, and S. Lee in “CFBOX:Superimposing 3D Human Face on Motion Picture”, in Proceedings of International Conference on Virtual Systems and Multimedia, 2001. Another example of avatar creation is shown by M. Lyons, A. Plante, S. Jehan, S. Inoue, and S. Akamatsu in “Avatar Creation using Automatic Face Recognition”, in Proceedings of ACM Multimedia 98, pages 427-434, 1998. U.S. Pat. No. 6,400,374 of Lanier disclosed a system for superimposing a foreground image like a human head with face to the background image.
In these approaches, the human face image essentially becomes the superimposing object to the background templates or pre-handled video image sequences. However, we can also superimpose other virtual objects onto the human face image. Human facial features, the smaller sub-objects on the face image, can provide the useful local coordinate information within the face image in order to augment the human facial image.
In the case of avatar creation, which is mentioned above, the captured face image and the motion of the user do not affect the visual result because the static template or pre-handled video sequence is a pre-defined static entity. In other words, the static background does not change or respond according to the user's motion. However, in the FET system, the superimposing virtual objects are attached to the user's face image and move along the face or body movement in real-time dynamically and automatically.
Conventionally, the facial feature detection has been one of the most researched topics in the computer vision community. One of the important facts of the facial feature detection resides in its practical applicability. It can be used for many computer vision applications, and it can also be used to support other computer vision algorithms, such as face recognition. Some examples of facial feature extraction applications are human face recognition, automated visual interpretation, security access control, low bit-rate video coding, automatic annotation for image database, and development of friendly human interfaces. U.S. Pat. No. 5,802,220 of Black et al., U.S. Pat. No. 5,852,669 of Eleftheriadis et al., and U.S. Pat. No. 6,381,345 of Swain disclosed systems for facial feature detection and tracking.
The FET system also uses the facial feature detection technique as a part of the modules within the system. However, it is not a simple facial feature detector nor facial feature tracker but a real-time facial image enhancement system. The FET system is a method and apparatus for enhancing facial images in a continuous video by superimposing virtual objects onto the facial images automatically and dynamically in real-time, comprising three main modules of initialization module, facial feature detection module, and superimposition module.